forum

Home / DeveloperSection / Forums / What are some of the most common optimization algorithms used to train deep learning models?

Ask Question
Utpal Vishwas

Utpal Vishwas

Follow
I am Utpal Vishwas from Uttar Pradesh. Have completed my B. Tech. course from MNNIT campus Prayagraj in 2022. I have good knowledge of computer networking.

Can you answer this question?

Answer

1 Answers

Krishnapriya Rajeev

Krishnapriya Rajeev

26-Apr-2023

Some of the most common and sophisticated optimization algorithms used to train deep learning models are:

  • Stochastic Gradient Descent (SGD): This is the most commonly used optimization algorithm in deep learning. It updates the model's parameters in the direction of the steepest gradient of the loss function.
  • AdaGrad: This algorithm adapts the learning rate for each parameter based on the historical gradient information, giving smaller learning rates to parameters associated with frequently occurring features.
  • RMSProp: This algorithm divides the learning rate by a running average of the magnitudes of recent gradients for each parameter, thereby making it less sensitive to noisy gradients.
  • Adam: This algorithm combines the ideas of momentum and AdaGrad, resulting in an adaptive learning rate that changes according to the momentums of gradients and a running estimate of the second moment of the gradients.
MindStick Training

Liked By